Knowledge Acquired by Foundation Models

Abstract During pre-training, a Foundation Model is trained on an extensive collection of documents and learns the distribution words in correct fluent language. In this chapter, we investigate knowledge acquired by PLMs larger Models. We first discuss application Models to specific benchmarks test large number areas examine if models are able derive conclusions from content. Another group tests assesses completing text applying probing classifiers that consider syntactic knowledge, semantic logical reasoning separately. Finally, reliable reproducible, i.e. whether they actually targeted properties yield same performance values when repeated other researchers.